Imaging System and Method for Use in Surgical and Interventional Medical Procedures

ABSTRACT

A system and method for displaying images of internal anatomy includes an image processing device configured to provide high resolution images of the surgical field from low resolution scans during the procedure. The image processing device digitally manipulates a previously-obtained high resolution baseline image to produce many representative images based on permutations of movement of the baseline image. During the procedure a representative image is selected having an acceptable degree of correlation to the new low resolution image. The selected representative image and the new image are merged to provide a higher resolution image of the surgical field. The image processing device is also configured to provide interactive movement of the displayed image based on movement of the imaging device, and to permit placement of annotations on the displayed image to facilitate communication between the radiology technician and the surgeon.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to provisional application No.61/390,488, filed on Oct. 6, 2010, and incorporates the entiredisclosure herein by reference.

BACKGROUND

The present invention contemplates a system and method for altering theway a patient image, such as by X-ray, is viewed and obtained. Moreparticularly, the inventive system and method provides means fordecreasing the overall radiation to which a patient is exposed during asurgical procedure but without significantly sacrificing the quality orresolution of the image obtained.

Many surgical procedures require obtaining an image of the patient'sinternal body structure, such as organs and bones. In some procedures,the surgery is accomplished with the assistance of periodic images ofthe surgical site. Surgery can broadly mean any invasive testing orintervention performed by medical personnel, such as surgeons,interventional radiologists, cardiologists, pain management physicians,and the like. In surgeries and interventions that are in effect guidedby serial imaging, which we will refer to as image guided, frequentpatient images are necessary for the physician's proper placement ofsurgical instruments, be they catheters, needles, instruments orimplants, or performance of certain medical procedures. Fluoroscopy, orfluoro, is one form of intraoperative X-ray and is taken by a fluorounit, also known as a C-arm. The C-arm sends X-ray beams through apatient and takes a picture of the anatomy in that area, such asskeletal and vascular structure. It is, like any picture, atwo-dimensional (2D) image of a three-dimensional (3D) space. However,like any picture taken with a camera, key 3D info may be present in the2D image based on what is in front of what and how big one thing isrelative to another.

A DRR is a digital representation of an X-ray made by taking a CT scanof a patient and simulating taking X-rays from different angles anddistances. The result is that any possible X-ray that can be taken forthat patient can be simulated, which is unique and specific to how thepatient's anatomical features look relative to one another. Because the“scene” is controlled, namely by controlling the virtual location of aC-Arm to the patient and the angle relative to one another, a picturecan be generated that should look like any X-ray taken in the operatingroom (OR).

Many imaging approaches, such as taking fluoro images, involve exposingthe patient to radiation, albeit in small doses. However, in these imageguided procedures, the number of small doses adds up so that the totalradiation exposure can be problematic not only to the patient but alsoto the surgeon or radiologist and others participating in the surgicalprocedure. There are various known ways to decrease the amount ofradiation exposure for a patient/surgeon when an image is taken, butthese approaches come at the cost of decreasing the resolution of theimage being obtained. For example, certain approaches use pulsed imagingas opposed to standard imaging, while other approaches involve manuallyaltering the exposure time or intensity. Narrowing the field of view canpotentially also decrease the area of radiation exposure and itsquantity (as well as alter the amount of radiation “scatter”) but againat the cost of lessening the information available to the surgeon whenmaking a medical decision. Further, often times images taken during asurgical intervention are blocked either by extraneous OR equipment orthe actual instruments/implants used to perform the intervention.Limiting the blocking of the normal anatomy behind those objects wouldhave tangible benefits to the medical community.

There is a need for a an imaging system, that can be used in connectionwith standard medical procedures, that reduces the radiation exposure tothe patient and medical personnel, but without any sacrifice in accuracyand resolution of an X-ray image. There is also a need for an imagingsystem that accounts for instruments and hardware, such as implants,that might otherwise obscure a full view of the surgical site.

SUMMARY

According to one aspect, a system and method is providing for generatinga display of a patient's internal anatomy for use in a surgical orinterventional medical procedure based on a previously acquired highresolution baseline image and a newly acquired low resolution image. Thehigh resolution image may be an image obtained during the procedure or apre-procedure image such as a DRR. The low resolution image may beacquired using a pulse or low radiation dose. The system contemplates animage processing device configured to digitally manipulate the highresolution baseline image to produce a baseline image set includingrepresentative images of the baseline image at a plurality ofpermutations of movements of the baseline image in 4D or 6D space. Thenew low resolution image is compared to the baseline image set to selecta representative image having an acceptable degree of correlation withthe new image. The image processing device may implement algorithms toperform the comparison, such as a principal component analysis or otherstatistical test. The image processing device is further configured tomerge the selected representative high resolution image with the new lowresolution image to generate a merged image to be displayed. The mergedimage may be further processed to allow alternating between the selectedhigh resolution image and the new low resolution image, or to adjust theamount that the two images are merged in the displayed image.

In another feature of the present disclosure, an imaging system mayinclude an image processing device that acts as a viewfinder as theimaging device is moved relative to the patient. In accordance with thisfeature, an image of the surgical field is acquired with the imagingdevice in a first orientation. That acquired image is continuouslydisplayed while the imaging device, patient or patient table is movedfrom the first orientation. This movement is tracked is used the imageprocessing device to move the displayed image in relation to the trackedmovement. With this feature, the display acts as a viewfinder to predicthow a new image would appear if captured at that time by the imagingdevice. This feature can thus be used to determine where the next liveimage of the patient's anatomy will be taken or can be used to assist institching multiple images together to form a larger panoramic view o thesurgical field. The image processing system may implement softwareadapted to optimize the predicted image and minimize misalignment or offangle appearance of the display. In another aspect, the image processingsystem permits annotation of the displayed image to identify anatomicfeatures or desired image trajectories or alignments.

In a further feature of the disclosed embodiments, a baseline image ofanatomy within a surgical field is acquired in a baseline orientation,and that baseline image is digitally manipulated to produce a baselineimage set including representative images of the baseline image at aplurality of permutations of movements of the baseline image. A newimage of the surgical field in which portions of the anatomy are blockedby objects. This new image is compared to the baseline image set toselect a representative image having an acceptable degree of correlationwith the new image. The image processing system generates a displayedimage showing the surgical field with the blocking objects minimized oreliminated. The system further permits fading the blocked objects in andout of the display.

DESCRIPTION OF THE FIGURES

FIG. 1 is a pictorial view of an image guided surgical setting includingan imaging system and an image processing device, as well as a trackingdevice.

FIG. 2 a is an image of a surgical field acquired using a full dose ofradiation in the imaging system.

FIG. 2 b is an image of the surgical field shown in FIG. 2 a in whichthe image was acquired using a lower dose of radiation.

FIG. 2 c is a merged image of the surgical field with the two imagesshown in FIGS. 2 a-b merged in accordance with one aspect of the presentdisclosure.

FIG. 3 is a flowchart of graphics processing steps undertaken by theimage processing device shown in FIG. 1.

FIG. 4 a is an image of a surgical field including an object blocking aportion of the anatomy.

FIG. 4 b is an image of the surgical field shown in FIG. 4 a with theimage of FIG. 4 a partially merged with a baseline image to display theblocked anatomy.

FIGS. 5 a-b are baseline and merged images of a surgical field includinga blocking object.

FIGS. 6 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of an in-boundsor out-of-bounds position of the imaging device for acquiring a newimage.

FIGS. 7 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of when a newimage can be stitched to a previously acquired image.

FIGS. 8 a-b are displays of the surgical field adjusted for movement ofthe imaging device or C-arm and providing an indicator of alignment ofthe imaging device with a desired trajectory for acquiring a new image.

FIG. 9 is a depiction of a display and user interface for the imageprocessing device shown in FIG. 1.

FIG. 10 is a graphical representation of an image alignment processaccording to the present disclosure.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the principles of theinvention, reference will now be made to the embodiments illustrated inthe drawings and described in the following written specification. It isunderstood that no limitation to the scope of the invention is therebyintended. It is further understood that the present invention includesany alterations and modifications to the illustrated embodiments andincludes further applications of the principles of the invention aswould normally occur to one skilled in the art to which this inventionpertains.

A typical imaging system 100 is shown in FIG. 1. The imaging systemincludes a base unit 102 supporting a C-arm imaging device 103. TheC-arm includes a radiation source 104 that is positioned beneath thepatient P and that directs a radiation beam upward to the receiver 105.It is known that the radiation beam emanated from the source 104 isconical so that the field of exposure may be varied by moving the sourcecloser to or away from the patient. The C-arm 103 may be rotated aboutthe patient P in the direction of the arrow 108 for different viewingangles of the surgical site. In some instances, implants or instrumentsT may be situated at the surgical site, necessitating a change inviewing angle for an unobstructed view of the site. Thus, the positionof the receiver relative to the patient, and more particularly relativeto the surgical site of interest, may change during a procedure asneeded by the surgeon or radiologist. Consequently, the receiver 105 mayinclude a tracking target 106 mounted thereto that allows tracking ofthe position of the C-arm using a tracking device 130. For instance, thetracking target 106 may include several infrared emitters spaced aroundthe target, while the tracking device is configured to triangulate theposition of the receiver 105 from the infrared signals emitted by theelement. The base unit 102 includes a control panel 110 through which aradiology technician can control the location of the C-arm, as well asthe radiation exposure. A typical control panel 110 thus permits thetechnician to “shoot a picture” of the surgical site at the surgeon'sdirection, control the radiation dose, and initiate a radiation pulseimage.

The receiver 105 of the C-arm 103 transmits image data to an imageprocessing device 122. The image processing device can include a digitalmemory associated therewith and a processor for executing digital andsoftware instructions. The image processing device may also incorporatea frame grabber that uses frame grabber technology to create a digitalimage for projection as displays 123, 124 on a display device 126. Thedisplays are positioned for interactive viewing by the surgeon duringthe procedure. The two displays may be used to show a images from twoviews, such as lateral and AP, or may show a baseline scan and a currentscan of the surgical site, or a current scan and a “merged” scan basedon a prior baseline scan and a low radiation current scan, as describedherein. An input device 125, such as a keyboard or a touch screen, canallow the surgeon to select and manipulate the on-screen images. It isunderstood that the input device may incorporate an array of keys ortouch screen icons corresponding to the various tasks and featuresimplemented by the image processing device 122. The image processingdevice includes a processor that converts the image data obtained fromthe receiver 105 into a digital format.

In one aspect of the present invention, the image processing device 122is configured to provide high quality real-time images on the displays123, 124 that are derived from lower detail images obtained using lowerdoses of radiation. By way of example, FIG. 2 a is a “full dose” (FD)x-ray image, while FIG. 2 b is a low dose and/or pulsed (LD) image ofthe same anatomy. It is apparent that the LD image is too “noisy” anddoes not provide enough information about the local anatomy for accurateimage guided surgery. While the FD image provides a crisp view of thesurgical site, the higher radiation dose makes taking multiple FD imagesduring a procedure highly problematic. Using the steps described herein,the surgeon is provided with a current image shown in FIG. 2 c thatsignificantly reduces the noise of the LD image, in some cases by about90%, so that surgeon is provided with a clear real-time image using alow dose of radiation. This capability allows for dramatically lessradiation exposure during the imaging to verify the position ofinstruments and implants during the procedure.

The flowchart of FIG. 3 depicts one embodiment of method according tothe present invention. In a first step 200, a baseline high resolutionFD image is acquired of the surgical site and stored in a memoryassociated with the image processing device. In some cases where theC-arm is moved during the procedure, multiple high resolution images canbe obtained at different locations in the surgical site, and then thesemultiple images “stitched” together to form a composite base image (asdiscussed below). Movement of the C-arm, and more particularly“tracking” the acquired image during these movements, is accounted forin other steps described in more detail herein. For the presentdiscussion it is assumed that the imaging system is relative fixed,meaning that only very limited movement of the C-arm and/or patient arecontemplated, such as might arise in an epidural pain procedure, spinalK-wire placement or stone extraction. The baseline image is projected instep 202 on the display 123 for verification that the surgical site isproperly centered within the image. In some cases, new FD images may beobtained until a suitable baseline image is obtained. In procedures inwhich the C-arm is moved, new baseline images are obtained at the newlocation of the imaging device, as discussed below. If the displayedimage is acceptable as a baseline image, a button may be depressed on auser interface, such as on the display device 126 or interface 125.

Once the baseline image is acquired, a baseline image set is generatedin step 204 in which the original baseline image is digitally rotated,translated and resized to create thousands of permutations of theoriginal baseline image. For instance, a typical two dimensional (2D)image of 128×128 pixels may be translated ±15 pixels in the x and ydirections at 1 pixel intervals, rotated ±9° at 3° intervals and scaledfrom 92.5% to 107.5% at 2.5% intervals (4 degrees of freedom, 4D),yielding 47,089 images in the baseline image set. (A three-dimensional(3D) image will imply a 6D solution space due to the addition of twoadditional rotations orthogonal to the x and y axis. An original CTimage data set can be used to form many thousands of DRRs in a similarfashion.) Thus, in this step, the original baseline image spawnsthousands of new image representations as if the original baseline imagewas acquired at each of the different movement permutations. This“solution space” may be stored in a graphics card memory of the imageprocessing device 122 in step 206 or formed as a new image is sent tothe GPU, depending on the number of images in the solution space and thespeed at which the GPU can produce those images. With current computingpower, on a free standing, medical grade computer, the generation ofthis baseline image set can occur in less than one second.

During the procedure, a new LD image is acquired in step 208, stored inthe memory associated with the image processing device, and projected ondisplay 123. Since the new image is obtained at a lower dose ofradiation it is very noisy. The present invention thus provides stepsfor “merging” the new image with an image from the baseline image set toproduce a clearer image on the second display 124 that conveys moreuseful information to the surgeon. The invention thus contemplates animage recognition or registration step 210 in which the new image iscompared to the images in the baseline image set to find a statisticallymeaningful match. A new “merged” image is generated in step 212 that maybe displayed on display 124 adjacent the view of the original new image.At various times throughout the procedure, a new baseline image may beobtained in step 216 that is used to generate a new baseline image setin step 204.

Step 210 contemplates comparing the current new image to the images inthe baseline image set. Since this step occurs during the surgicalprocedure, time and accuracy are critical. Preferably, the step canobtain an image registration in less than one second so that there is nomeaningful delay between when the image is taken by the C-arm and whenthe merged image is displayed on the device 126. Various algorithms maybe employed that may be dependent on various factors, such as the numberof images in the baseline image set, the size and speed of the computerprocessor or graphics processor performing the algorithm calculations,the time allotted to perform the computations, and the size of theimages being compared (e.g., 128×128 pixels, 1024×1024 pixels, etc). Inone approach, comparisons are made between pixels at predeterminedlocations described above in a grid pattern throughout 4D space. Inanother heuristic approach, pixel comparisons can be concentrated inregions of the images believed to provide a greater likelihood of arelevant match. These regions may be “pre-seeded” based on knowledgefrom a grid or PCA search (defined below) or data from a tracking system(such as an optical surgical navigation device) or location data fromthe DICOM file or the equivalent.

In another approach, a principal component analysis (PCA) is performed,which can allow for comparison to a larger number of larger images inthe allotted amount of time than is permitted with the full resolutiongrid approach. In the PCA approach, a determination is made as to howeach pixel of the image set co-varies with each other. A covariancematrix may be generated using only a small portion of the total solutionset—for instance, a randomly selected 10% of the baseline image set.Each image from the baseline image set is converted to a column vector.In one example, a 70×40 pixel image becomes a 2800×1 vector. Thesecolumn vectors are normalized to a mean of 0 and a variance of 1 andcombined into a larger matrix. The covariance matrix is determined fromthis larger matrix and the largest eigenvectors are selected. For thisparticular example, it has been found that 30 PCA vectors can explainabout 80% of the variance of the respective images. Thus, each 2800×1image vector can be multiplied by a 2800×30 PCA vector to yield a 1×30vector. The same steps are applied to the new image—the new image isconverted to a 2800×1 image vector and multiplication with the 2800×30PCA vector produces a 1/30 vector corresponding to the new image. Thesolution set (baseline image) vectors and the new image vector arenormalized and the dot product of the new image vector to each vector inthe solution space is calculated. The solution space baseline imagevector that yields the largest dot product (i.e., closest to 1) isdetermined to be the closest image to the new image. It is understoodthat the present example may be altered with different image sizesand/or different principal components used for the analysis. It isfurther understood that other known techniques may be implemented thatmay utilize eigenvectors, singular value determination, mean squarederror, mean absolute error, and edge detection, for instance. It isfurther contemplated that various image recognition approaches can beapplied to selected regions of the images or that various statisticalmeasures may be applied to find matches falling within a suitableconfidence threshold. A confidence or correlation value may be assignedthat quantifies the degree of correlation between the new image and theselected baseline image, or selected ones of the baseline image set, andthis confidence value may be displayed for the surgeon's review. Thesurgeon can decide whether the confidence value is acceptable for theparticular display and whether another image should be acquired.

In the image guided surgical procedures, tools, implants and instrumentswill inevitably appear in the image field. These objects are typicallyradiodense and consequently block the relevant patient anatomy fromview. The new image obtained in step 210 will thus include an artifactof the tool T that will not correlate to any of the baseline image set.The presence of the tool in the image thus ensures that the comparisontechniques described above will not produce a high degree ofregistration between the new image and any of the baseline image set.Nevertheless, if the end result of each of the above procedures is toseek out the highest degree of correlation, which is statisticallyrelevant or which exceeds a certain threshold, the image registrationmay be conducted with the entire new image, tool artifact and all.

Alternatively, the image registration steps may be modified to accountfor the tool artifacts on the new image. In one approach, the new imagemay be evaluated to determine the number of image pixels that are“blocked” by the tool. This evaluation can involve comparing a grayscalevalue for each pixel to a threshold and excluding pixels that falloutside that threshold. For instance, if the pixel grayscale values varyfrom 0 (completely blocked) to 10 (completely transparent), a thresholdof 3 may be applied to eliminate certain pixels from evaluation.Additionally, when location data is available for various tracked tools,algorithmically areas that are blocked can be mathematically avoided.

Once the image registration is complete, the new image may be displayedwith the selected image from the baseline image set in different ways.In one approach, the two images are merged, as illustrated in FIGS. 4 a,b. The original new image is shown in FIG. 4 a with the instrument Tplainly visible and blocking the underlying anatomy. A partially mergedimage generated in step 212 (FIG. 3) is shown in FIG. 4 b in which theinstrument T is still visible but substantially mitigated and theunderlying anatomy is visible. The two images may be merged by combiningthe digital representation of the images in a conventional manner, suchas by adding or averaging pixel data for the two images. In oneembodiment, the surgeon may identify a specific region of interest inthe displayed image, such as through the user interface 125, and themerging operation can be configured to utilize the baseline image datafor the display outside the region of interest and conduct the mergingoperation for the display within the region of interest. The userinterface 125 may be provided with a “slider” that controls the amountthe baseline image versus the new image that is displayed in the mergedimage. In another approach, the surgeon may alternate between thecorrelated baseline image and the new image or merged image, as shown inFIGS. 5 a, b. The image in FIG. 5 a is the image from the baseline imageset found to have the highest degree of correlation to the new image.The image in FIG. 5 b is the new image obtained. The surgeon mayalternate between these views to get a clearer view of the underlyinganatomy and a view of the current field with the instrumentation T,which in effect by alternating images digitally removes the instrumentfrom the field of view, clarifying its location relative to the anatomyblocked by it.

As indicated above, the present invention also contemplates a surgicalnavigation procedure in which the imaging device or C-arm 103 is moved.Thus, the present invention contemplates tracking the position of theC-arm rather than tracking the position of the surgical instruments andimplants as in traditional surgical navigation techniques, usingcommercially available tracking devices or the DICOM information fromthe imaging device. Tracking the C-arm requires a degree of accuracythat is much less than the accuracy required to track the instrumentsand implants. In this embodiment, the image processing device 122receives tracking information from the tracking device 130. The objectof this aspect of the invention is to ensure that the surgeon sees animage that is consistent with the actual surgical site regardless of theorientation of the C-arm relative to the patient.

Tracking the position of the C-arm can account for “drift”, which is agradual misalignment of the physical space and the imaging (or virtual)space. This “drift” can occur because of subtle patient movements,inadvertent contact with the table or imaging device and even gravity.This misalignment is often visually imperceptible, but can generatenoticeable shifts in the image viewed by the surgeon. These shifts canbe problematic when the surgical navigation procedure is being performed(and a physician is relying on the information obtained from thisdevice) or when alignment of new to baseline images is required toimprove image clarity. The use of image processing eliminates theinevitable misalignment of baseline and new images. The image processingdevice 122 further may incorporate a calibration mode in which thecurrent image of the anatomy is compared to the predicted image. Thedifference between the predicted and actual movement of the image can beaccounted for by an inaccurate knowledge of the “center of mass” or COM,described below, and drift. Once a few images are obtained and the COMis accurately established, recalibration of the system can occurautomatically with each successive image taken and thereby eliminatingthe impact of drift.

The image processing device 122 may operate in a “tracking mode” inwhich the movement of the C-arm is monitored and the currently displayedimage is moved accordingly. The currently displayed image may be themost recent baseline image, a new LD image or a merged image generatedas described above. This image remains on one of the displays 123, 124until a new picture is taken by the imaging device 100. This image isshifted on the display to match the movement of the C-arm using theposition data acquired by the tracking device 130. A tracking circle 240may be shown on the display, as depicted in FIGS. 6 a, 6 b. The trackingcircle identifies an “in bounds” location for the image. When thetracking circle appears in red, the image that would be obtained withthe current C-arm position would be “out of bounds” in relation to abaseline image position, as shown in FIG. 6 a. As the C-arm is moved bythe radiology technician the representative image on the display ismoved. When the image moves “in bounds”, as shown in FIG. 6 b, thetracking circle 240 turns green so that the technician has an immediateindication that the C-arm is now in a proper position for obtaining anew image. The tracking circle may be used by the technician to guidethe movements of the C-arm during the surgical procedure. The trackingcircle may also be used to assist the technician in preparing a baselinestitched image. Thus, an image position that is not properly aligned forstitching to another image, as depicted in FIG. 7 a, will have a redtracking circle 240, while a properly aligned image position, as shownin FIG. 7 b, will have a green tracking circle. The technician can thenacquire the image to form part of the baseline stitched image.

The present invention contemplates a feature that enhances thecommunication between the surgeon and the radiology technician. Duringthe course of a procedure the surgeon may request images at particularlocations or orientations. One example is what is known as a “Fergusonview” in spinal procedures in which an AP oriented C-arm is canted toalign directly over a vertebral end plate with the end plate oriented“flat” or essentially parallel with the beam axis of the C-arm.Obtaining a Ferguson view requires rotating the C-arm or the patienttable while obtaining multiple AP views of the spine, which iscumbersome and inaccurate using current techniques, requiring a numberof fluoroscopic images to be performed to find the one best aligned tothe endplate. The present invention allows the surgeon to overlay a gridonto a single image or stitched image and provide labels for anatomicfeatures that can then be used by the technician to orient the C-arm.Thus, as shown in FIG. 8 a, the image processing device 122 isconfigured to allow the surgeon to place a grid 245 within the trackingcircle 240 overlaid onto a Lateral image. The surgeon may also locatelabels 250 identifying anatomic structure, in this case spinalvertebrae. In this particular example, the goal is to align the L2-L3disc space with the center grid line 246. To assist the technician, atrajectory arrow 255 is overlaid onto the image to indicate thetrajectory of an image acquired with the C-arm in the current position.As the C-arm moves, changing orientation off of pure AP, the imageprocessing device evaluates the C-arm position data obtained from thetracking device 230 to determine the new orientation for trajectoryarrow 255. The trajectory arrow thus moves with the C-arm so that whenit is aligned with the center grid line 246, as shown in FIG. 8 b, thetechnician can shoot the image knowing that the C-arm is properlyaligned to obtain a Ferguson view along the L3 endplate. Thus,monitoring the lateral view until it is rotated and centered along thecenter grid line allows the radiology technician to find the AP Fergusonangle without guessing and taking a number of incorrect images.

The image processing device may be further configured to show thelateral and AP views simultaneously on respective displays 123 and 124,as depicted in FIG. 9. Either or both views may incorporate the grid,labels and trajectory arrows. This same lateral view may appear on thecontrol panel 110 for the imaging system 100 for viewing by thetechnician. As the C-arm is moved to align the trajectory arrow with thecenter grid line, as described above, both the lateral and AP images aremoved accordingly so that the surgeon has an immediate perception ofwhat the new image will look like. Again, once the technician properlyorients the C-arm, as indicated by alignment of the trajectory arrowwith the center grid line, a new AP image is acquired. As shown in FIG.9, a view may include multiple trajectory arrows, each aligned with aparticular disc space. For instance, the uppermost trajectory arrow isaligned with the L1-L2 disc space, while the lowermost arrow is alignedwith the L5-S1 disc space. In multiple level procedures the surgeon mayrequire a Ferguson view of different levels, which can be easilyobtained by requesting the technician to align the C-arm with aparticular trajectory arrow.

In another feature, a radiodense asymmetric shape can be placed in aknown location on the C-arm detector. This creates the ability to linkthe coordinate frame of the C-arm to the arbitrary orientation of theC-arm's image coordinate frame. As the C-arm's display may be modifiedto generate an image having any rotation or mirroring, detecting thisshape radically simplifies the process of image comparison and imagestitching. Thus, as shown in FIG. 10, the baseline image B includes theindicia “K” at the 9 o'clock position of the image. The new image N isobtained in which the indicia has been rotated by the physician ortechnologist away from the default orientation. Comparing this new imageto the baseline image set is unlikely to produce any registrationbetween images due to this angular offset. In one embodiment, the imageprocessing device detects the actual rotation of the C-arm from thebaseline orientation while in another embodiment the image processingdevice uses image recognition software to locate the “K” indicia in thenew image and determine the angular offset from the default position.This angular offset is used to alter the rotation and/or mirror imagethe baseline image set. The baseline image selected in the imageregistration step 210 is maintained in its transformed orientation to bemerged with the newly acquired image. This transformation can includerotation and mirror-imaging, to eliminate the display effect that ispresent on a C-arm.

In another aspect, it is known that as the C-arm radiation source 104moves closer to the table, the size of the image captured by thereceiver 105 becomes larger; moving the receiver closer to the tableresults in a decrease in image size. Whereas the amount that the imagescales with movements towards and away from the body can be easilydetermined, if the C-arm is translated along the table, the image willshift, with the magnitude of that change depending upon the proximity ofthe “center of mass” (COM) of the patient to the radiation source.Although the imaged anatomy is of 3D structures, with a high degree ofaccuracy, mathematically we can represent this anatomy as a 2D pictureof the 3D anatomy placed at the COM of the structures. Then, forinstance, when the COM is close to the radiation source, small movementswill cause the resulting image to shift greatly. Until the COM isdetermined, though, the calculated amount that the objects on the screenshift will be proportional to but not equal to their actual movement.The difference is used to calculate the actual location of the COM. TheCOM is adjusted based on the amount that those differ, moving it awayfrom the radiation source when the image shifted too much, and theopposite if the image shifts too little. The COM is initially assumed tobe centered on the table to which the reference arc of the trackingdevice is attached. The true location of the COM is fairly accuratelydetermined using the initial two or three images taken during initialset-up of the imaging system, and reconfirmed/adjusted with each newimage taken. Once the COM is determined in global space, the movement ofthe C-arm relative to the COM can be calculated and applied to translatethe baseline image set accordingly for image registration.

The image processing device 122 may also be configured to allow thesurgeon to introduce other tracked elements into an image, to help guidethe surgeon during the procedure. A closed-loop feedback approach allowsthe surgeon to confirm that the location of this perceived trackedelement and the image taken of that element correspond. Specifically,the live x-ray and the determined position from the surgical navigationsystem are compared. In the same fashion that knowledge of the baselineimage, through image recognition, can be used to track the patient'sanatomy even if blocked by radiodense objects, knowledge of theradiodense objects, when the image taken is compared to their trackedlocation, can be used to confirm their tracking When both theinstrument/implant and the C-arm are tracked, the location of theanatomy relative to the imaging source and the location of the equipmentrelative to the imaging source are known. This information can thus beused to quickly and interactively ascertain the location of theequipment or hardware relative to the anatomy. This feature can, by wayof example, have particular applicability to following the path of acatheter in an angio procedure, for instance. In a typical angioprocedure, a cine, or continuous fluoro, is used to follow the travel ofthe catheter along a vessel. The present invention allows intersplicingpreviously generated images of the anatomy with the virtual depiction ofthe catheter with live fluoro shots of the anatomy and actual catheter.Thus, rather than taking 15 fluoro shots per second for a typical cineprocedure, the present invention allows the radiology technician to takeonly one shot per second to effectively and accurately track thecatheter as it travels along the vessel. The previously generated imagesare spliced in to account for the fluoro shots that are not taken. Thevirtual representations can be verified to the live shot when taken andrecalibrated if necessary.

In certain procedures it is possible to fix the position of the vascularanatomy to larger features, such as nearby bones. This can beaccomplished using DRRs from prior CT angiograms (CTA) or from actualangiograms taken in the course of the procedure. Either, approach may beused as a means to link angiograms back to bony anatomy and vice versa.To describe in greater detail, the same CTA may be used to producedifferent DRRs, such as DRRs highlighting just the bony anatomy andanother in a matched set that includes the vascular anatomy along withthe bones. A baseline fluoro image taken of the patient's bony anatomycan then be compared with the bone DRRs to determine the best match.Instead of displaying the result using bone only DRR, the matched DRRthat includes the vascular anatomy can be used to merge with the newimage. In this approach, the bones help to place the radiographicposition of the catheter to its location within the vascular anatomy.Since it is not necessary to continually image the vessel itself, as thepicture of this structure can be overlaid onto the bone only imageobtained, the use of contrast dye can be limited versus prior proceduresin which the contrast dye is necessary to constantly see the vessels.

Following are examples of specific procedures utilizing the features ofthe image processing device discussed above. These are just a fewexamples as to how the software can be manipulated using differentcombinations of baseline image types, display options, and radiationdosing and not meant to be an exhaustive list.

Pulsed New Image/Alternated with/Baseline of FD Fluoro or PreoperativeX-Ray

A pulsed image is taken and compared with a previously obtained baselineimage set containing higher resolution non-pulsed image(s) taken priorto the surgical procedure. Registration between the current image andone of the baseline solution set provides a baseline image reflectingthe current position and view of the anatomy. The new image isalternately displayed or overlaid with the registered baseline image,showing the current information overlaid and alternating with the lessobscured or clearer image.

Pulsed New Image/Alternated with/Baseline Derived from DRR

A pulsed image is taken and compared with a previously obtained solutionset of baseline images, containing higher resolution DRR obtained from aCT scan. The DRR image can be limited to just show the bony anatomy, asopposed to the other obscuring information that frequently “cloud” afilm taken in the OR (e.g.—bovie cords, EKG leads, etc.) as well asobjects that obscure bony clarity (e.g.—bowel gas, organs, etc.). Aswith the above example, the new image that is registered with one of theprior DRR images, and these images are alternated or overlaid on thedisplay 123, 124.

Pulsed New Image/Merged Instead of Alternated

All of the techniques described above can be applied and instead ofalternating the new and registered baseline images, the prior andcurrent image are merged. By performing a weighted average or similarmerging technique, a single image can be obtained which shows both thecurrent information (e.g.—placement of instruments, implants, catheters,etc.) in reference to the anatomy, merged with a higher resolutionpicture of the anatomy. In one example, multiple views of the merger ofthe two images can be provided, ranging from 100% pulsed image to 100%DRR image. A slide button on the user interface 125 allows the surgeonto adjust this merger range as desired.

New Image is a Small Segment of a Larger Baseline Image Set

The imaging taken at any given time contains limited information, a partof the whole body part. Collimation, for example, lowers the overalltissue radiation exposure and lowers the radiation scatter towardsphysicians but at the cost of limiting the field of view of the imageobtained. Showing the actual last projected image within the context ofa larger image (e.g.—obtained prior, preoperatively or intraoperatively,or derived from CTs)—merged or alternated in the correction location—cansupplement the information about the smaller image area to allow forincorporation into reference to the larger body structure(s). The sameimage registration techniques are applied as described above, exceptthat the registration is applied to a smaller field within the baselineimages (stitched or not) corresponding to the area of view in the newimage.

Same as Above, Located at Junctional or Blocked Areas

Not infrequently, especially in areas that have different overalldensities (e.g.—chest vs. adjacent abdomen, head/neck/cervical spine vs.upper thorax), the area of an x-ray that can be clearly visualized isonly part of the actual image obtained. This can be frustrating to thephysician when it limits the ability to place the narrow view into thelarger context of the body or when the area that needs to be evaluatedis in the obscured part of the image. By stitching together multipleimages, each taken in a localized ideal environment, a larger image canbe obtained. Further, the current image can be added into the largercontext (as described above) to fill in the part of the image clouded byits relative location.

Unblocking the Hidden Anatomy or Mitigating its Local Effects

As described above, the image processing device performs the imageregistration steps between the current new image and a baseline imageset that, in effect, limits the misinformation imparted by noise, be itin the form of x-ray scatter or small blocking objects (e.g.—cords,etc.) or even larger objects (e.g.—tools, instrumentation, etc.). Inmany cases, it is that part of the anatomic image that is being blockedby a tool or instrument that is of upmost importance to the surgerybeing performed. By eliminating the blocking objects from the image thesurgery becomes safer and more efficacious and the physician becomesempowered to continue with improved knowledge. Using an image that istaken prior to the noise being added (e.g.—old films, baseline single FDimages, stitched together fluoro shots taken prior to surgery, etc.) oridealized (e.g.—DRRs generated from CT data), displaying that prior“clean” image, either merged or alternated with the current image, willmake those objects disappear from the image or become shadows ratherthan dense objects. If these are tracked objects, then the blocked areacan be further deemphasized or the information from it can be eliminatedas the mathematical comparison is being performed, further improving thespeed and accuracy of the comparison.

The image processing device configured as described herein providesthree general features that (1) reduce the amount of radiation exposurerequired for acceptable live images, (2) provide images to the surgeonthat can facilitate the surgical procedure, and (3) improve thecommunication between the radiology technician and the surgeon. Withrespect to the aspect of reducing the radiation exposure, the presentinvention permits low dose images to be taken throughout the surgicalprocedure and fills in the gaps created by “noise” in the current imageto produce a composite or merged image of the current field of view withthe detail of a full dose image. In practice this allows for highlyusable, high quality images of the patient's anatomy generated with anorder of magnitude reduction in radiation exposure than standard FDimaging using unmodified features present on all common, commerciallyavailable C-arms. The techniques for image registration described hereincan be implemented in a graphic processing unit and can occur in asecond or so to be truly interactive; when required such as in CINEmode, image registration can occur multiple times per second. A userinterface allows the surgeon to determine the level of confidencerequired for acquiring registered image and gives the surgeon options onthe nature of the display, ranging from side-by-side views to fadein/out merged views.

With respect to the feature of providing images to the surgeon thatfacilitate the surgical procedure, several digital imaging techniquescan be used to improve the user's experience. One example is an imagetracking feature that can be used to maintain the image displayed to thesurgeon in an essentially a “stationary” position regardless of anyposition changes that may occur between image captures. In accordancewith this feature, the baseline image can be fixed in space and newimages adjust to it rather than the converse. When successive images aretaken during a step in a procedure each new image can be stabilizedrelative to the prior images so that the particular object of interest(e.g.—anatomy or instrument) is kept stationary in successive views. Forexample, as sequential images are taken as a bone screw is introducedinto a body part, the body part remains stationary on the display screenso that the actual progress of the screw can be directly observed.

In another aspect of this feature, the current image including blockingobjects can be compared to earlier images without any blocking objects.In the registration process, the image processing device can generate amerged image between new image and baseline image that deemphasizes theblocking nature of the object from the displayed image. The userinterface also provides the physician with the capability to fade theblocking object in and out of the displayed view.

In other embodiments in which the object itself is being tracked, avirtual version of the blocking object can be added back to thedisplayed image. The image processing device can obtain position datafrom a tracking device following the position of the blocking object anduse that position data to determine the proper location and orientationof the virtual object in the displayed image. The virtual object may beapplied to a baseline image to be compared with a new current image toserve as a check step—if the new image matches the generated image (bothtool and anatomy) within a given tolerance then the surgery can proceed.If the match is poor, the surgery can be stopped (in the case ofautomated surgery) and/or recalibration can take place. This allows fora closed-loop feedback feature to facilitate the safety of automation ofmedical intervention.

For certain procedures, such as a pseudo-angio procedure, projecting thevessels from a baseline image onto current image can allow a physicianto watch a tool (e.g.—micro-catheter, stent, etc.) as it travels throughthe vasculature while using much less contrast medium load. The adjacentbony anatomy serves as the “anchor” for the vessels—the bone isessentially tracked, through the image registration process, and thevessel is assumed to stay adjacent to this structure. In other words,when the anatomy moves between successive images, the new image isregistered to a different one of the baseline image set that correspondsto the new position of the “background” anatomy. The vessels from adifferent but already linked baseline image containing the vascularstructures can then be overlaid or merged with the displayed image whichlacks contrast. If necessary or desired, intermittent angios can betaken to confirm. When combined with a tracked catheter, a workingknowledge of the location of the instrument can be included into theimages. A cine (continuous movie loop of fluoro shots commonly used whenan angiogram is obtained) can be created in which generated images areinterspliced into the cine images, allowing for many fewer x-rays to beobtained while an angiogram is being performed or a catheter is beingplaced. Ultimately, once images have been linked to the originalbaseline image, any of these may be used to merge into a current image,producing a means to monitor movement of implants, the formation ofconstructs, the placement of stents, etc.

In the third feature—improving communication—the image processing devicedescribed herein allows the surgeon to annotate an image in a mannerthat can help guide the technician in the positioning of the C-arm as tohow and where to take a new picture. Thus, the user interface 125 of theimage processing device 122 provides a vehicle for the surgeon to add agrid to the displayed image, label anatomic structures and identifytrajectories for alignment of the imaging device. As the technicianmoves the imaging device or C-arm, the displayed image is moved. Thisfeature allows the radiology tech to center the anatomy that is desiredto be imaged in the center of the screen, at the desired orientation,without taking multiple images each time the C-arm is brought back inthe field to obtain this. This feature provides a view finder for theC-arm, a feature lacking currently. The technician can activate theC-arm to take a new image with a view tailored to meet the surgeon'sexpressed need.

In addition, linking the movements of the C-arm to the images takenusing DICOM data or a surgical navigation backbone, for example, helpsto move the displayed image as the C-arm is moved in preparation for asubsequent image acquisition. “In bound” and “out of bounds” indicatorscan provide an immediate indication to the technician whether a currentmovement of the C-arm would result in an image that cannot be correlatedor registered with any baseline image, or that cannot be stitchedtogether with other images to form a composite field of view. The imageprocessing device thus provides image displays that allow the surgeonand technician to visualize the effect of a proposed change in locationand trajectory of the c-arm. Moreover, the image processing device mayhelp the physician, for instance, alter the position of the table or theangle of the C-arm so that the anatomy is aligned properly (such asparallel or perpendicular to the surgical table). The image processingdevice can also determine the center of mass (COM) of the exact centerof an x-rayed object using two or more x-ray shots from two or moredifferent gantry angles/positions, and then use this COM information toimprove the linking of the physical space (in millimeters) to thedisplayed imaging space (in pixels).

The image recognition component disclosed herein can overcome the lackof knowledge of the location of the next image to be taken, whichprovides a number of benefits. Knowing roughly where the new image iscentered relative to the baseline can limit the need to scan a largerarea of the imaging space and, therefore, significantly increase thespeed of image recognition software. Greater amounts of radiationreduction (and therefore noise) can be tolerated, as there exists aninternal check on the image recognition. Multiple features that aremanual in the system designed without surgical navigation, such asbaseline image creation, switching between multiple baseline image sets,and stitching, can be automated. These features are equally useful in animage tracking context.

As described above, the systems and methods correlate or synchronize thepreviously obtained images with the live images to ensure that anaccurate view of the surgical site, anatomy and hardware, is presentedto the surgeon. In an optimum case, the previously obtained images arefrom the particular patient and are obtained near in time to thesurgical procedure. However, in some cases no such prior image isavailable. In such cases, the “previously obtained image” can beextracted from a database of CT and DRR images. The anatomy of mostpatients is relatively uniform depending on the height and stature ofthe patient. From a large database of images there is a high likelihoodthat a prior image or images of a patient having substantially similaranatomy can be obtained. The image or images can be correlated to thecurrent imaging device location and view, via software implemented bythe image processing device 122, to determine if the prior image issufficiently close to the anatomy of the present patient to reliablyserve as the “previously obtained image” to be interspliced with thelive images.

The display in FIG. 9 is indicative of the type of display and userinterface that may be incorporated into the image processing device 122,user interface 125 and display device 126. For instance, the displaydevice may include the two displays 122, 123 with “radio” buttons oricons around the perimeter of the display. The icons may be touch screenbuttons to activate the particular feature, such as the “label”, “grid”and “trajectory” features shown in the display. Activating a touchscreen or radio button can access a different screen or pull down menuthat can be used by the surgeon to conduct the particular activity. Forinstance, activating the “label” button may access a pull down menu withthe labels “L1”, “L2”, etc., and a drag and drop feature that allows thesurgeon to place the labels at a desire location on the image. The sameprocess may be used for placing the grid and trajectory arrows shown inFIG. 9.

While the invention has been illustrated and described in detail in thedrawings and foregoing description, the same should be considered asillustrative and not restrictive in character. It is understood thatonly the preferred embodiments have been presented and that all changes,modifications and further applications that come within the spirit ofthe invention are desired to be protected.

What is claimed is:
 1. A method for generating a display of an image ofa patient's internal anatomy in a surgical field during a medicalprocedure, comprising: acquiring a high resolution baseline image of thesurgical field including the patient's internal anatomy in a baselineorientation; digitally manipulating the high resolution baseline imageto produce a baseline image set including representative images of thebaseline image at a plurality of permutations of movements of thebaseline image from the baseline orientation; acquiring a new image ofthe surgical field at a lower resolution; comparing the new image to therepresentative images in the baseline image set and selecting therepresentative image having an acceptable degree of correlation with thenew image; and merging the selected representative image with the newimage and displaying the merged image.
 2. The method of claim 1, whereinthe baseline image is selected from the group including a pre-procedurefull dose fluoroscopic image, a DRR and a CT scan image.
 3. The methodof claim 1, wherein the new image is one of a pulse or low dose image.4. The method of claim 1, wherein: in the step of digitally manipulatingthe high resolution image the permutations of movements form apredefined grid of image movements; and the step of comparing the newimage to the representative images of the baseline image set includescomparing overlapping pixels between the representative image and thenew image.
 5. The method of claim 1, wherein the step of comparing thenew image to the representative images of the baseline image setincludes: performing a principal component analysis (PCA) on the pixelsof the representative images in the baseline image set to generate oneor more PCA vectors; producing a PCA matrix of PCA vectors for eachpixel in a representative image; generating a column vector for eachrepresentative image and the new image of pixel data for each pixel inthe image; performing a matrix multiplication of the PCA matrix and eachcolumn vector to generate a new column vector for each representativeimage and the new image; obtaining the dot product of the column vectorfor the new image and the column vector for each of the representativeimage; and selecting a representative image for which the dot product iswithin a pre-determined threshold.
 6. The method of claim 1, in whichthe medical procedure includes tools, instruments, implants or otherobjects that block or obscure the internal anatomy in an image of thesurgical field, wherein the step of comparing the new image to therepresentative images of the baseline image set includes only comparingportions of the images outside the portions that are blocked orobscured.
 7. The method of claim 6, wherein the location of the blockedor obscured portions of the new image are determined by determiningwhich pixels have a value outside a pre-determined threshold.
 8. Themethod of claim 1, wherein: the step of digitally manipulating the highresolution baseline image includes providing parallel images to eachrepresentative image in which the image of certain anatomic features isreduced or enhanced; and the step of merging the selected representativeimage includes merging and displaying the parallel image to the selectedrepresentative image.
 9. An image processing device for generating adisplay of an image of a patient's internal anatomy during a medicalprocedure, comprising: a memory for storing a high resolution baselineimage of the surgical field including the patient's internal anatomy ina baseline orientation and a new image of the surgical field at a lowresolution; and a processor configured to; digitally manipulate the highresolution baseline image to produce a baseline image set includingrepresentative images of the baseline image at a plurality ofpermutations of movements of the baseline image from the baselineorientation; perform software instructions for comparing the new imageto the representative images in the baseline image set and selecting therepresentative image having an acceptable degree of correlation with thenew image; digitally merge the selected representative image with thenew image; and generate signals for displaying the merged image on adisplay device.
 10. The image processing device of claim 9, wherein: theprocessor is further configured to digitally manipulate the highresolution image such that the permutations of movements form apredefined grid of image movements; and the software instructions forcomparing the new image to the representative images of the baselineimage set include instructions for comparing overlapping pixels betweenthe representative image and the new image.
 11. The image processingdevice of claim 9, wherein the software instructions for comparing thenew image to the representative images of the baseline image set includeinstructions for: performing a principal component analysis (PCA) on thepixels of the representative images in the baseline image set togenerate one or more PCA vectors; producing a PCA matrix of PCA vectorsfor each pixel in a representative image; generating a column vector foreach representative image and the new image of pixel data for each pixelin the image; performing a matrix multiplication of the PCA matrix andeach column vector to generate a new column vector for eachrepresentative image and the new image; obtaining the dot product of thecolumn vector for the new image and the column vector for each of therepresentative image; and selecting a representative image for which thedot product is within a pre-determined threshold.
 12. The imageprocessing device of claim 9, in which the medical procedure includestools, instruments, implants or other objects that block or obscure theinternal anatomy in an image of the surgical field, wherein the softwareinstructions for comparing the new image to the representative images ofthe baseline image set include instructions for only comparing portionsof the images outside the portions that are blocked or obscured.
 13. Theimage processing device of claim 12, wherein the processor is configuredto determine the location of the blocked or obscured portions of the newimage by determining which pixels of the representative images have avalue outside a pre-determined threshold.
 14. The image processingdevice of claim 9, further comprising a user interface operable to allowmanual adjustment of the degree of digitally merging the selectedrepresentative image with the new image.
 15. The image processing deviceof claim 14, wherein: the user interface is further operable to allowmanually switching between a display of one or more of therepresentative image, the new image and the merged image; and theprocessor generates signals for displaying on a display device accordingto the user interface.
 16. A method for generating a display of an imageof a patient's internal anatomy in a surgical field during a medicalprocedure, comprising: acquiring an image of the surgical field with theimaging device in a first orientation; displaying the acquired image;moving the imaging device, patient, or table from the first orientation;tracking the movement of the imaging device, patient, or table from thefirst orientation; moving the displayed image in relation to the trackedmovement prior to acquiring a new image of the surgical field with theimaging device; and overlaying indicia indicative of a desired movementof the displayed image.
 17. The method of claim 16, wherein the indiciais a grid overlaid on the displayed image that remains stationaryrelative to the displayed image as the displayed images moves.
 18. Themethod of claim 17, wherein the indicia includes a trajectory indicatorindicative of the direction of view for the new image that moves withthe displayed image.
 19. The method of claim 18, wherein the imagingdevice is moved until the trajectory indicator is aligned with part ofthe grid prior to acquiring the new image.
 20. The method of claim 16,further comprising the step of overlaying identifiers corresponding tothe anatomic features in the displayed image that move with thedisplayed image.
 21. The method of claim 16, in which the medicalprocedure is a surgical navigation procedure in which the position of anobject, such as a tool or instrument, is tracked, wherein the new imageincludes an image of the tool or instrument; and the method furthercomprises: introducing a representation of the object on the acquiredimage; after moving the displayed image in relation to the trackedmovement, acquiring position data corresponding to the tracked positionof the object and comparing the position data with the position of theobject on the moved image; and recalibrating the moved image based onthe comparison of the position data with the position of the object onthe moved image.
 22. A method for generating a display of an image of apatient's internal anatomy in a surgical field during a medicalprocedure, in which the medical procedure includes tools, instruments,implants or other objects that block or obscure the internal anatomy inan image of the surgical field, comprising: acquiring a high resolutionbaseline image of anatomy within the surgical field in a baselineorientation; digitally manipulating the high resolution baseline imageto produce a baseline image set including representative images of thebaseline image at a plurality of permutations of movements of thebaseline image from the baseline orientation; acquiring a new image ofthe surgical field in which portions of the anatomy are blocked byobjects; comparing the new image to the representative images in thebaseline image set to select a representative image having an acceptabledegree of correlation with the new image; and displaying the selectedrepresentative image to show the surgical field minimizing oreliminating the blocking objects.
 23. The method of claim 22, whereinthe step of comparing the new image to the representative images of thebaseline image set includes only comparing portions of the imagesoutside the portions that are blocked or obscured.
 24. The method ofclaim 23, wherein the locations of the blocked or obscured portions ofthe new image are determined by determining which pixels have a valueoutside a pre-determined threshold.